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Title I of the Elementary and Secondary Education Act has become 
the largest Federal elementary and secondary education program in the 
history of the United States, with expenditures in fiscal year 1981 
of approximately $3.2 billion. The guestion of whether Title I "works" 
has been debated since the program was initiated in 1965, and has inten- 
sified in recent months due to proposed cuts in funding for the program. 

The question of whether Title I works does not lead to a simple 
answer, since the program has many different kinds of objectives, ano 
hence can be either effective or ineffective in different ways. For 
example, Title I can be examined in terms of who receives services, 
in terms of how well developed those services are, in terms of 
how cost-ef fecientl / programs are run, or (and this will focus the 
basis of the discussion in this paper) in terms of how effective 
services are in enhancing the educational achievement of participants. 

When discussing whether or not Title I "works," another point must 
be kept in mind there is no national Title I program per se. Title I 
is a source o f funding for local school districts, each of which designs 
and implements its own Title I proaram, within federal and State guidelines. 
Thus, at the national level one is measuring only the aggregate, overall 
effects of 14,000 different programs, while at the local level one has 
the opportunity to examine closely the educational (and other) effects 
of a particular program, in a particular place, with particular types 
of participants. 

Numerous studies have attempted to measure the effectiveness of 
Title I. The most recent of these national efforts are the Sustaining 
Effects Study and the implementation of the Title I Evaluation and 
Reporting System. In addition to those two efforts, the National Asessment 
of Educational Progress (NAEP) has analyzed achievement in Title I schools, 
though they are not able to identify Title I participants and examine 
their achievement explicitly. In addition, State and local education 
agencies conduct Title I evaluations beyond those reguired to complete 
the Title I Evaluation and Reporting System forms. Indeed, the Title I 
law requires districts to conduct evaluation for program improvement and 
decision-makinq as v»ell as for reporting achievement information. 

In this paper, we will review the findings from each of these sources 
of information, in order to develop different perspectives on assessing 
the effectiveness of Title I. While in the ideal world (or the world 
of trial by jury) most of the evidence will point toward a "most likely" 
explanation, that kind of overall finding regarding the effectiveness 
of Title I has remained somewhat elusive. 




Sources of Information 



The Sustaining Effects Study 

The Sustaining Effects Study (SES) is a six year study of the 
Title I, Part A (Grants to Local Education Agencies) programs which was 
bequn in 1975 and is essentially completed. The study was desiqned to 
document: (1) the characteristics of districts, schools, and students 
participating in Title I; (2) the nature of Title I services provided to 
proqram participants; and (3) an analysis of the effectiveness of those 
services over several years of proqram participation. 

The SES represents the most comprehensive effort to date in attempting 
to document and analyze the nature and effects of compensatory education 
programs. Data were collected on all students in a nationally representative 
sample of over 200 elementary schools during the three year period from 
the fall of 1975 through the spring of 1978. Not only were data collected 
on certain characteristics of students, teachers and principals, on 
student progress in the basic skills and on attitudes towards school, but 
interviews were conducted with the parents of a representative sub-sample 
of 15,000 students. Information on the effects of student achievement 
over one year and over three years is now available. 

Title I Evaluation and Reporting System 

The 1979-80 school year marked the first year in which all States 
provided information using the Title I Evaluation and Reporting System 
(TIERS), a system of models and reporting forms designed to yipld aqgreqatabl 
nationwide information about Title I participation and effects. Reports 
were received from all of the States by the late summer of 1981, and 
included achievement information for reading, mathematics, and languaqe 
arts programs in qrades 2 through 12, as well as student participation 
and instru ctional staffing information for all grades. 



State summary data . States had the option of reporting data from 
all of their local education agencies (LEAs) or from a sample of one-third 
of their LEAs. Twenty States reported on a sample of their LEAs, and 
their test data were weighted for the data analyses. States reported, 
for each grade, test cycle, and subject the following information: 
membership (the enrollment for the project on a particular day), the 
number of students with pretest and posttest scores, the weighted mean 
posttest score, and the weighted mean gain. In addition to this 
achievement information, States provided information describing 
the types of services they provided, characteristics of staff and 
participants, training activities, etc. This information was 
summarized across districts by each State. 

Project level d ata. In addition to the aggregate information, 
States reported project level information for grades 2, 6, and 10. 
Project level information included an LEA identification code, a 
descriptor of the type of project, subject area, evaluation model, test 
interval (whether fall-spring, fall-fall, or spring-spring testinq), 
hours per week of instruction, total number of hours of project instruction, 
student-instructor ratio, membership, number of students with pre- and 
posttest scores, posttest identification code, mean posttest score and 
mean gain.* 

1 Expressed in normal curve equivalents, or NCEs , an equal-interval 

unit with a mean of 50 and a standard deviation of 21.06. 
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The TIERS State aggregate data should be interpreted with some 
caution, since these data: (1) are based on scores aggregated across 
many different tests, of varying quality and appropriateness, and admin- 
istered under only locally controlled conditions; (2) in some situations 
were obtained in States or districts implementing the evaluation pro- 
cedures for the first time; and (3) in some situations, quality control 
procedures may not have been adequately implemented. In addition, Linn, 
Dunbar, Harnisch, and Hastings (draft, 1982) have recently conducted 
an assessment of the validity of evaluations using the TIERS models, 
and have concluded that: (1) fall-to-spring gain estimates are subject 
to numerous sources of bias, the cumulative effects of which make the 
estimated gains overstate the amount of impact; (2) while annual testing 
produces results less succeptible to bias (and reduces test burden), 
there is still aDt to be a small positive bias of 1, or possibly 2, 
NCEs; and (3) Models b and C and the non-normed variation of Model A 
tend to have practical limitations, technical inadequacies, or both* 

Test scores were reported in NCEs, with the fall-to-spring (6 month) 
evaluations aggregated separately from the annual (12 month, most commonly 
sprinq to spring) evaluations. Projects were aggregated across the 
model employed; however, estimates based on the qrades 2, 6, and 10 
project level information indicate that approximately 98% of the students 
were tested with Model A-l, the norm-referenced model. 

While achievement data were collected from all SEAs (which includes 
Insular areas, Territories jnd the Bureau of Indian Affairs), this 
paper includes data only from the 50 States and the District of Columbia. 
The Insular areas and Territories are significantly dissimilar to the 
States (and to each other) to exclude them from a national analysis 
(e.g., due to lack of availability of suitable tests and norms, testing 
in languages other than English). 

National Assessment- of Educational Progress 

The National Assessment of Educational Progress (NAEP) conducts 
periodic assessments of student performance in a wide variety of areas. 
In a national assessment of reading (NAEP, 1981), the performance of 
9-, 13-, and 17-year olds was compared at three points in time, spanning 
9 years. Information was available on which schools in the sample were 
eligible for Title I funds, but not on which students were in Title I 
Droqrams. 

State and Local Title I Evaluations 

Numerous State and locsl education agencies have shared the results 
of their own extensive Title I evaluations with ED. While these studies 
are not nationally representative, they do point to types of projects 
which may be having a positive impact on student achievement. State and 
local evaluations often focus on ancillary questions, such as whether 
Title I summer programs are effective or whether or not gains that 
were observed just following Title I project participation are sustained 
ovor time. 
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Types of Comparisons 

A primary qoal of Title I evaluation is to determine whether 
students learn more with the supplementary services provided with Title 
I funds tran they would have learned without the supplementary services. 
Classical experimental desiqns that would allow for equivalent comparison 
and treatment qroups are virtually impossible to implement, leaving 
evaluators faced with the necessity of using surroqate comparison groups, 
using serendipitous and probably non-comparable comparison qroups, 
usinq statistical estimates based on non-comparable groups, or attempting 
to work from baseline data and infer changes in performance to the program. 
The main reason for these constraints stems from the legislative require- 
ment that Title I serve the educat ional ly neediest children, within 
eligible schools. When Title I becomes Chapter 1 (of the Education 
Consolidation and Improvement Act) on October 1, 1982, this constraint 
may change somewhat, since Chapter 1 (accordinq to the interpretation 
of the law in a recent Notice of Proposed Rulemaking) may no lonqer 
strictly require that only those most in educational need be served. 

The sources of information listed above each use some combination 
of these methodolcqies in order to arrive at estimates of proqram effect. 
The TIERS provides three models for evaluation of Title I projects. As 
has been noted, virtually all students are evaluated with the norm-reTer^nced 
model, Model A, which uses a surrogate comparison group consisting of 
students with similar achievement standing who were in the norming sample 
for the particular achievement test used. The SES, in some of their 
analyses, employed a comparison group composed of students similar to 
Title I students who were in schools not served by Title I. In other 
analyses, the SES also adopted a norm-referenced approach. NAEP 
uses baseline data to measure change over time. While each of these 
methods is different, one can be optimistic (at least to start with) 
that the results may converge. 

Educational effects can also be measured over varyinq lengths of 
time. It is preferable, given the difficulty in assessing educational 
impact, to measure students over a substantial lenqth of time; however, 
practical considerations may make this difficult or impossible. Never- 
theless, while lonq-term evaluation may be preferable, if the long term 
impact of a program is positive one would expect to see positive (and 
probably smaller) impacts over shorter periods. The TIERS assesses 
achievement impact over relatively short periods of time either 6 
months (October to April in a typical fall-to-sprinq evaluation) or 12 
months (April to April in a typical spring-to-sprinq evaluation). The 
SES looked et student achievement level both over the fall to soring 
of a typical district Title I evaluation and over periods of time up to 
three years. The NAEP studies look at achievement of 9-, 13-, and 17- 
year-olds at app r oximate four year intervals. 

Short-term Achievement Impact 

Short -term achievement impact, i.e., information on achievement over 
one year or less, is available both from the SES and from the TIERS. 
Information over both a full year and over the six months typical of a 
fall-to-spring evaluation are available from both sources. Overall, it 
was found that compensatory services, particularly Title r services, 
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often have positive impacts on achievement. Different patterns of 
achievement were found for different qroups, however. In addition, 
information from the TIERS shows that students tested on an annual 
test^nq schedule consistently showed much smaller gains than those tested 
fal 1 -to-sprinq. 

The results of the TIERS reading and mathematics data are summarized 
in Tables 1 and 2. The results of the fall-to-spring evaluations showed 
substantial qains for Title I students — durinq six months in the program, 
the typical elementary school reading student moved from the 16th to the 
25th percentile. Numerous studies and local evaluations have questioned 
whether students who show such gains on a fall-to-spring testing schedule 
show similar qains when measured over a full year. Early studies attributed 
the differences to "summer loss", speculating that low achievinq students 
tended to forqet what they had learned over the summer. Later work, 
however, has suggested that the fault may lie with the methodoloqy, and 
that several sources of bias may combine to qreatly overstate the estimated 
gains. In virtually all cases where both fall-spring and spring-spring 
testinq cycles have both been implemented (within grade levels within 
States), it is almost always tne C3se that sprinq posttest scores e~e 
comparable, while fall pretest scores are substantially lower than 
pretest scores obtained the previous spring. 

Given these concerns and observations, it may be wise to focus on 
the annual results rather than the fall-to-spring results. For both 
readinq and mathematics, the results show modest positive gains at the 
elementary school level and a mixed pattern of qains and losses at the 
high school level. The typical elementary school readinq student would 
move from the 22nd to the 26th percentile in achievement. Research has 
indicated, however, that the results of Model Al have a bias of approximately 
+1.0 NCE (RMC, 1982; Linn et al , 1982), so a correction of that 
amount may be in order. Table 3 presents the corrected achievement 
results for readinq and mathematics. 

In readinq, small positive qains are found throughout the elementary 
qrades except at grade 2; results for the high school grades continue to 
be mixed. In mathematics, the pattern is mixed, with no effect found at 
qrades 2 and 12; losses in grades 3, 9, 10, and 11; and qains in the 
other grades. In both readinq and mathematics, there is a tendency for 
larger gains to be found in the upper elementary grades than in the 
lower elementary qrades. 

The SES, which compared compensatory education students with a 
similar qroup of needy children who were not served by compensatory 
education, found a different pattern of results. In the SES, positive 
effects in reading were found at grades 1, 2, and 3, but not in grades 
4, 5, and 6. In mathematics, the SES found positive effects at qrades 
1 through 6. The TIERS found similar results before correction for 
bias, althouqh the gain at grade 3 was very small. After correction for 
bias, the TIERS still noted gains in grades 4 to 6 as well as in grades 7 
and 8 (the SES covered only qrades 1 - 6). 

Both studies indicate however, that compensatory education, particu- 
larly Title I, have positive effects on achievement, althouqh the effects 
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qenerally are modest. The TIERS sugqests that Title I is more likely 
to be effective at the elerentary grades than at hi qh school. While this 
may indicate that remediation is more effective with younqer students, it 
should also be noted that the population served at the hi qh school level 
is considerably needier that at the elementary level. At the elementary 
level, the typical Title I student scores at the 22nd percentile in 
reading; the typical student in qrade 11 is at the 14th percentile. 
In addition, standardized tests (particularly in reading) may be quite 
insensitive to the high school curriculum, since it is quite often the 
case that no raw score gain at all (or a gain of cne or two points, or 
even a loss of one or two points) will not change a student's percentile 
status from pre- to posttesting. 

It may be the case (this is discussed more fully later in the paper) 
that Title I students who are only "sliqhtly needy" may, with the extra 
help received through Title I, qain enouqh to enable them to function well 
in the regular classroom and to maintain their qains, while the "severely 
needy" students are unable to make such qains. Thus, at the elementary 
grades, where a greater proportion of students are served, and thus more 
"sliqhtly needy" students are served, qains are noted, while at the 
upper qrades, where only "severely needy" students are served, no gains 
are found. 



Table 1 

1979-80 Title I Reading Achievement Results (50 States plus the District of Columbia) 



Annual Testing Schedule F a 1 1-to-Sprinq Testing Schedule • 

Grade Weiqhted Nornal Curve Percentile Percent^ Weighted Normal Curve Percentile Percent 

Number Equivalent Additional Number Equivalent Additions 





Tested 


Pre 


Post 


Gai n 


Pre 


Post 


Growth 


Tested 


Pre 


Post 


Gai n 


Pre 


Post 


Growth 


2 


85,019 


37.6 


38.6 


1.0 


28 


29 


4 


310,555 


30.8 


40.2 


9.4 


18 


32 


77 


3 


108,708 


34.3 


36.7 


2.4 


23 


26 


17 


293,909 


28.7 


36.1 


7.4 


16 


26 


90 


4 


108,576 


34.7 


36.6 


1.9 


23 


26 


20 


270,826 


28.7 


35.6 


7.0 


16 


25 


111 


5 


112,387 


33.9 


36.2 


2 " 


22 


26 


32 


246,159 


29.4 


35.5 


6.1 


16 


25 


132 


6 


107,706 


33.9 


37.2 


3.2 


22 


27 


42 


212,819 


29.7 


35.7 


6.0 


17 


25 


158 


7 


66,923 


33.9 


35.8 


1 .8 


22 


25 


27 


152,417 


28.8 


34.3 


5.5 


16 


23 


124 


8 


58,026 


33.6 


35.8 


2.2 


22 


25 


31 


122,013 


29.0 


34.0 


5.0 


16 


22 


113 


9 


30,082 


32.0 


33.8 


1 .8 


20 


22 


38 


66,475 


28.3 


33.5 


5.2 


15 


22 


163 


10 


14,215 


30.2 


29.5 


-0.7 


17 


17 


-16 


36,102 


28.6 


32.8 


4.2 


16 


21 


131 




8,579 


27.5 


25.3 


-2.2 


14 


12 


-43 


1 7,734 


27.3 


30.5 


3.2 


14 


18 


123 


12 


7,146 


25.4 


26.8 


1.4 


12 


14 


33 


8,383 


25.6 


30.0 


4.4 


12 


17 


133 



2 The percent additional qrowth measure reoresents achievement gain 
made by Title I participants over and above that which is typically 
made durinq the course of a school year by children scoring at pretest 
time the same as the Title I children, but who do not receive supplementary 
services. A percent additional growth measure of 100% means that 
the Title I children have qained twice as much on a test as their 
peers who began the year at the same perce n tile. 
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Table 2 

1979-80 Title I Mathematics Achievement Results (50 States plus the District of Columbia) 



Annual Testing Schedule 

Grade Weiqhted Normal Curve Percenti 1 e Percent Weiqhted Normal Curve 

Number Equivalent Additional Number Equivalent 

Tested We Post Gain Pre Post Growth Tested "Pre Post Gain 



Fal 1 -to-Spr i ng Test ing Schedule 

Percenti 1 e Percent 

Additions 
Pre Post Growth 



2 


50,084 


41.9 


43.0 


1.1 


35 


37 


5 


124,576 


32.0 


42.5 


10.5 


20 


36 


88 


3 


65,407 


39.7 


40.1 


0.4 


31 


32 


0.2 


137,608 


31.5 


40.1 


8.6 


19 


32 


69 


4 


70,637 


37.5 


39.2 


1.8 


28 


30 


15 


147,333 


30.8 


39.8 


9.0 


18 


31 


134 


5 


71 ,030 


36.6 


39.0 


2.5 


26 


30 


23 


136,872 


30.5 


38.7 


8.2 


18 


30 


115 


6 


69,002 


35.4 


39.3 


3.9 


24 


31 


44 


119,003 


30.9 


38.6 


7.7 


18 


29 


141 


7 


36,268 


34.5 


36.7 


2.2 


23 


26 


29 


74,807 


30.6 


36.9 


6.3 


18 


27 


150 


8 


29,530 


34.3 


37.1 


2.8 


23 


27 


44 


60,747 


30.1 


36.3 


6.2 


17 


26 


184 


9 


15,971 


34.6 


35.1 


0.5 


23 


24 


10 


28,579 


29.8 


35.9 


6.2 


17 


25 


200 


10 


7,718 


32.9 


31.6 


-1.4 


21 


19 


-34 


12,192 


32.0 


37.3 


5.3 


20 


27 


204 


11 


4,158 


34.9 


35.3 


0.4 


24 


24 


11 


5,270 


32.5 


38.1 


5.6 


20 


29 


311 


12 


3,587 


33.8 


34.9 


1.0 


22 


24 


48 


2,195 


30.7 


37.2 


6.5 


18 


27 


650 



Table 3 

1979-80 Title I Reading and Mathematics Annual 
Achievement Results Corrected for Bias 



Grade 



~NCTT 



Reading 



Mathematics 



Pre Corrected 



Percenti le 
Pre Corrected 



NCE 

We Corrected 



Percenti le 
"Pre Corrected 







Post 


Gai n 




Post 




Post 


Gain 




Post 


2 


37.6 


37.6 


0.0 


28 


28 


41.9 


42.0 


0.1 


35 


35 


3 


34.3 


35.7 


1.4 


23 


25 


39.7 


39.1 


-0.6 


31 


30 


4 


34.7 


35.6 


0.9 


23 


25 


37.5 


38.2 


0.8 


28 


29 


5 


33.9 


35.2 


1.3 


22 


24 


36.6 


38.0 


1.5 


26 


28 


6 


33.9 


36.7 


2.2 


22 


26 


35.4 


38.3 


2.9 


24 


29 


7 


33.9 


34.8 


0.8 


22 


24 


34.5 


35.7 


1.2 


23 


25 


8 


33.6 


34.8 


1.2 


22 


24 


34.3 


36.1 


1.8 


23 


26 


9 


32.0 


32.8 


0.8 


20 


21 


34.6 


34.1 


-0.5 


23 


23 


10 


30.2 


28.5 


-1.7 


17 


15 


32.9 


30.6 


-2.4 


21 


18 


11 


27.5 


24.3 


-3.2 


14 


11 


34.9 


34.3 


-0.6 


24 


23 


12 


25.4 


25.8 


0.4 


12 


13 


33.8 


33.9 


0.0 


22 


22 



Longitudinal Effects of Title I 

The SES contains information on Title I students in qrades i - 6 over 
three years, during which time students participated in various combina- 
tions of Title I and reaular programs. Figure 1 presents achievement 
information for four groups of students: 
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o Regular students who did not need and did not participate in 
Title I. 

o Title I participants in Year 1 who 'graduated out 1 of Title I 
in Year 2 and stayed out. 

o Title I participants in Years 1 and 2 who graduated out' of 
Title I in Year 3. 

o Title I participants who participated in the proqram for all 
three years. 

The dotted line seqments in Fiqure 1 indicate Title I participation 
across tine. An examination of these lines shows that: 

o In reading, Title I graduates do not generally fall back noticeably 
after participation, ceases. 

o In math, there generally is a decline after participation ceases. 

Three year participants do not show improvement over the three years, which 
is to be expected—had they shown substantial improvement, they would have 
been graduated out of the program. This does indicate, however, that some 
students do not show gains even after multiple years of compensatory 
education. 
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Fiqure 1. Effect: of "graduation" from Title I on student achievement. 



9 

One other piece of evidence for the effectiveness of Title I is the 
NAEP studies which document improvements in the educational status of 
minority qroup nine-year-olds over the past four years, and which 
also show improved achievement levels in Title I eligibile schools. 
While the studies do not identify Title I children, it least partially 
these achievement gains may be attributed to increased attention to 
basic skills and to effective compensatory edu:ation programs such as 
Title I. 

In summary, it appears that Title 1 has sone sustained benefits for 
program participants (particularly in reading) but that some stuaents 
(particularly in math) regress when Title I support is removed. Growth 
in practical skills, as measured by an instrument developed specifically 
for the SES, was found not to be a benefit from participation in compen- 
satory education proqrams, and the relationships between attitudes and 
participation are unclear. 

These results contradict somewhat findinqs from Drevious studies, 
and they confirm others. In particular, early reports from the SES 
(Interim Report, 1980 and Report 11, 1981) compared the achievement 
levels of regular students, Titl^ I 'graduates,' continuing Title I 
students, and students who were no lonqer in the proqram because of 
promotion to a qrade with no services or because the school lost fundinq. 
The Title I graduates' were compared to those students whose services 
were continuous over two years. The qrowth rate of students who 
■qraduated' was higher than tho. qrowth rate of students still roceivinq 
services, leading to the conclusion that termination of services aid 
not result in students disastrously falling back to the level of 
continuinq students. This is a different situation, however, than when 
students are compared over time, since the students who "qraduated" 
^ay have always had a hiqher achievement level than the continuinq 
students. 

The indication that certain subqroups of students may perform 
better than others is interestinq, especially given the TIERS findinq 
that qains are found at the eTementary school level but not at the hiqh 
school level, where a smaller, more needier, proportion of students are 
served. One can note from Fiqure 1 that in reading, the students who 
made achievement gains, and who maintained those qains, beqan at a hiqher 
achievement level than did the students who did not show qains in the 
proqram. 
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MEASURING THE FFFECTS OF COMPENSATORY EDUCATION - 
SEARCHING FOR CONVERGENCE FROM NATIONAL, STATE AND LOCAL EVALUATIONS 

Judith I. Anderson 
Robert M. Stonehil 1 
U.S. Department of Education 

liie U.S. Department of Education (ED) has implemented two systems of 
evaluation to examine the nature and effectiveness of ESEA Title I programs 
nationwide. The first strategy involves the sponsoring of national studies, 
the most important and extensive of which is the Study of the Sustaining Effects 
of Compensatory Education (the "Sustaining Effects Study"). The second strategy 
utilizes a system of aggregating locally implemented evaluations at the state 
and ultimately the Federal level. The Title I Evaluation and Reporting System 
(TIERS) is a set of models and reporting procedures developed by ED and 
supported by a technical assistance network. TIERS has been«£ul1y implemented 
nationally for the first time in 1979-80, 

While the information provided through the two strategies is collected 
in very different ways, e.g. TIERS is universally implemented while studies 
only sample a small portion of projects; TIERS allows for the use of multiple 
instruments while national studies usually adopt one particular test, the 
basic questions addressed are the same — what types of services are provided 
and to whom, and how effective are the services at improving educational 
achievement? The purpose of this presentation is to compare and contrast 
the evaluation strategies and the results, with particular emphasis on the 
TIERS, the Sustaining Effects Study and the results of the recent National 
Assessment longitudinal study. 
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